Stochastic processes with random contexts: a characterization, and adaptive estimators for the transition probabilities

This paper introduces the concept of random context representations for the transition probabilities of a finitealphabet stochastic process. Processes with these representations generalize context tree processes (a.k.a. variable length Markov chains), and are proven to coincide with processes whose transition probabilities are almost surely continuous functions of the (infinite) past. This is similar to a classical result by Kalikow about continuous transition probabilities. Existence and uniqueness of a minimal random context representation are shown, in the sense that there exists a unique representation that “looks into the past as little as possible” in order to determine the next symbol. Both this representation and the transition probabilities can be consistently estimated from data, and some finite sample adaptivity properties are also obtained (including an oracle inequality). In particular, the estimator achieves minimax performance, up to logarithmic factors, for the class of binary renewal processes whose arrival distributions have bounded moments of order 2 + γ. Index Terms Stochastic processes, estimation, statistics.